Controlling display performance using display statistics and feedback

ABSTRACT

One embodiment provides for a display system to generate and display data on a display device, the display system comprising one or more graphics processors to generate one or more frames of data for display on the display device; a window manager to submit a request to display the one or more frames of data; a display engine to present the one or more frames of data to the display device for display; and display logic to receive the request to display the one or more frames of data and generate one or more display events for the display engine based on the request to display the one or more frames of data, wherein the display logic is to manage a set of statistics associated with the request.

CROSS-REFERENCE

The present application is a continuation of U.S. application Ser. No.15/274,658 filed Sep. 23, 2016, which is a non-provisional applicationclaiming the benefit of pending U.S. Provisional Application No.62/345,539 filed on Jun. 3, 2016, and U.S. Provisional Application Nos.62/348,622, 62/348,626, 62/348,631, all filed Jun. 10, 2016, which arehereby incorporated herein by reference.

BACKGROUND OF THE DISCLOSURE

Legacy display systems operate at a constant refresh rate in which adisplay image is updated at a fixed number of cycles per second. Thisfixed display refresh rate can create playback issues when displayingcertain content. For example, a display panel having a 60 Hz refreshrate exhibit frame judder when viewing a 24 Hz movie, as the panelrefreshes every 16.6 ms, whereas the frame is generated using 41.6 mssample times, which is an uneven (e.g., 2.5) multiple of the frameduration. Accordingly, some frames are scanned by the display twicewhile others are scanned three times.

While some legacy displays allow the panel refresh rate to be reduced toa fixed refresh rate, visual quality may suffer. For example, blur maybe introduced into animated or moving content when the display refreshrate is too low for the displayed content. Moving high spatial frequencyimages across the panel, such as text displayed while scrolling a webpage or text document may appear blurry when refreshed at a low rate. Ingeneral, the higher the refresh rate of a display device, the sharperdisplayed animation becomes. Thus, a moving image displayed on a displayhaving a refresh rate of 120 Hz may yield a visibly sharper image andmay allow the text on the scrolling image to be legible undercircumstances where such text would be blurred and illegible at lowerrefresh rates.

Additionally, higher refresh displays provide benefits other thantemporal visual quality. For example, for touch input displays, thelatency between a touch input and displaying a result of the touch inputmay ultimately be gated by how quickly a new, updated frame can bepresented on the display. For example, if a display refreshes at 60 Hz,then a new frame generated based on a touch input may have to wait untilthe beginning of the next 16 ms period. Provided that a new frame isready for display, an 120 Hz display reduces the response time by halfof that of a 60 Hz display.

However, in most devices, particularly mobile and handheld devices, thedisplay hardware consumes a tremendous amount of the overall energybudget of the device. Increasing the refresh rate may further increasethe already large energy demands of operating the display. For example,in some displays, operating the display at 120 Hz instead of thestandard 60 Hz generally supported by displays may result in up to a150% increase in display power requirements. Thus, for a mobile device,a high display refresh can have a significant negative impact onbattery-based operating time.

SUMMARY OF THE DESCRIPTION

Various embodiments are described to enable the improved visual qualityprovide by higher refresh rates while balancing the additional energyrequirements of increased refresh rates.

One embodiment provides for a display system to generate and displaydata on a display device, the display system comprising one or moregraphics processors to generate one or more frames of data for displayon the display device; display logic to receive a request to display theone or more frames of data, the request including a requestedpresentation time in which the one or more frames of data are to bedisplayed; and a display engine to present the one or more frames ofdata to the display device for display at a target presentation time,the target presentation time derived from the requested presentationtime, wherein the display engine is to adjust a refresh rate of thedisplay device based on the target presentation time of the one or moreframes of data.

One embodiment provides for a non-transitory machine-readable mediumstoring instructions which, when executed by one or more processors,cause the one or more processors to perform operations to generate anddisplay data on a display device, the operations comprising receiving arequest to display one or more frames of data on a display device, theone or more frames of data generated via the one or more processors,wherein the request includes a requested presentation time in which theone or more frames of data are to be displayed; deriving a targetpresentation time based on the requested presentation time, the targetpresentation time derived based on a programming time for a displayengine; programming the display engine to present the one or more framesof data to the display device for display; and adjusting a refresh rateof the display device based on the target presentation time of the oneor more frames of data.

One embodiment provides for an electronic device comprising a variablerefresh rate display device; one or more graphics processors to generateone or more frames of data for display on the variable refresh ratedisplay device; display logic to receive a request to display the one ormore frames of data, the request including a requested presentation timein which the one or more frames of data are to be displayed; and adisplay engine to present the one or more frames of data to the variablerefresh rate display device for display at a target presentation time,the target presentation time derived from the requested presentationtime, wherein the display engine is to adjust a refresh rate of thevariable refresh rate display device based on the target presentationtime of the one or more frames of data.

One embodiment provides for a display system to generate and displaydata on a display device, the display system comprising one or moregraphics processors to generate one or more frames of data for displayon the display device; a display engine to present the one or moreframes of data to the display device for display; display logic toreceive a request to display the one or more frames of data and generateone or more display events for the display engine based on the request,wherein the request includes a first hint to indicate a complexity ofthe one or more frames of data and the one or more display eventsincludes a second hint derived from the first hint; and a performancecontroller to adjust one or more performance related parameters for thedisplay system based on the second hint.

One embodiment provides for a non-transitory machine readable mediumstoring instructions which, when executed by one or more processors,cause the one or more processors to perform operations comprisingreceiving a request to display one or more frames of data on a displaydevice of a display system, the request including a first hint toindicate a complexity of the one or more frames of data; generating oneor more display events based on the request, the one or more displayevents including a second hint derived from the first hint; andsignaling a performance controller associated with the one or moreprocessors to adjust one or more performance related parameters of theone or more processors based on the second hint.

One embodiment provides for an electronic device comprising a displaydevice having a variable refresh rate; one or more graphics processorsto generate one or more frames of data for display on the displaydevice; a display engine to present the one or more frames of data tothe display device for display; display logic to receive a request todisplay the one or more frames of data and generate one or more displayevents for the display engine based on the request, wherein the requestincludes a first hint to indicate a complexity of the one or more framesof data and the one or more display events includes a second hintderived from the first hint; and a performance controller to adjust oneor more performance related parameters for the display system based onthe second hint.

One embodiment provides for a display system to generate and displaydata on a display device, the display system comprising one or moregraphics processors to generate one or more frames of data for displayon the display device; a window manager to submit a request to displaythe one or more frames of data; a display engine to present the one ormore frames of data to the display device for display; and display logicto receive the request to display the one or more frames of data andgenerate one or more display events for the display engine based on therequest to display the one or more frames of data, wherein the displaylogic is to manage a set of statistics associated with the request andperiodically submit the set of statistics to the window manager, the setof statistics including a requested presentation time for a frame and anactual display time in which the frame is displayed on the displaydevice.

One embodiment provides for a non-transitory machine readable mediumstoring instructions which, when executed by one or more processors ofan electronic device, cause the one or more processors to performoperations comprising generating one or more frames of data at a firstframe rate; submitting a request to display logic to display the one ormore frames of data; receiving the request at the display logic andgenerating one or more display events for a display engine based on therequest; processing the display events at the display engine to displaythe one or more frames of data; generating a set of statisticsassociated with processing the display events, the set of statisticsincluding a requested presentation time for a frame and an actualdisplay time in which the frame is displayed on the display device; andadjusting the first frame rate to a second frame rate based on the setof statistics.

One embodiment provides for an electronic device comprising a displaydevice having a variable refresh rate; one or more graphics processorsto generate one or more frames of data for display on the displaydevice; a window manager to submit a request to display the one or moreframes of data; a display engine to present the one or more frames ofdata to the display device for display; and display logic to receive therequest to display the one or more frames of data and generate one ormore display events for the display engine based on the request todisplay the one or more frames of data, wherein the display logic is tomanage a set of statistics associated with the request and periodicallysubmit the set of statistics to the window manager, the set ofstatistics including a requested presentation time for a frame and anactual display time in which the frame is displayed on the displaydevice.

One embodiment provides for a method of generating and displaying dataon a display device, the method comprising receiving an input tointeract with an image presented via the display; generating a sequenceof frames in response to the input, the sequence of frames having ananimation frame rate independent of a current refresh rate of thedisplay device, the animation frame rate determined based on a parameterassociated with the input; configuring display events associated withthe sequence of frames, the display events including configuration datato configure a display engine to display one or more frames in thesequence of frames, each frame in the sequence of frames having arequested presentation time based on the animation frame rate;processing a first display event associated with a first frame in thesequence of frames, wherein processing the first display event includesreading the configuration data to configure the display engine todisplay the first frame; presenting the configuration data to thedisplay engine at a display time based on the requested presentationtime; and dynamically refreshing the display device based on the displaytime to display the sequence of frames.

One embodiment provides for a non-transitory machine readable mediumstoring instructions which, when executed by one or more processors,cause the one or more processors to perform operations comprisingreceiving an input to interact with an image presented via a displaydevice; generating a sequence of frames in response to the input, thesequence of frames having an animation frame rate independent of acurrent refresh rate of the display device, the animation frame ratedetermined based on a parameter associated with the input; configuringdisplay events associated with the sequence of frames, the displayevents including configuration data to configure a display engine todisplay one or more frames in the sequence of frames, each frame in thesequence of frames having a requested presentation time based on theanimation frame rate; processing a first display event associated with afirst frame in the sequence of frames, the processing including readingthe configuration data to configure the display engine to display thefirst frame; presenting the configuration data to the display engine ata display time based on the requested presentation time; and dynamicallyrefreshing the display device based on the display time to display thesequence of frames.

One embodiment provides for a display system comprising a display deviceto display a first frame including a first image; an input device toreceive a sequence of inputs to interact with the first image displayedwithin the first frame; one or more graphics processors to generate asecond image for display within a second frame and a third image fordisplay within a third frame, the first frame having a first requestedpresentation time and the third frame having a second requestedpresentation time; and a display engine to present the second frame fordisplay at the first requested presentation time, present the thirdframe for display at the second requested presentation time, and causethe display device to dynamically refresh to display the second frameand the third frame, wherein a refresh rate of the display variesbetween display of the first frame, second frame, and third frame.

The above summary does not include an exhaustive list of all aspects ofthe present invention. It is contemplated that the invention includesall systems and methods that can be practiced from all suitablecombinations of the various aspects summarized above, and also thosedisclosed in the Detailed Description below.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention is illustrated by way of example, and notlimitation, in the figures of the accompanying drawings in which likereferences indicate similar elements, and in which:

FIG. 1 is a block diagram of a graphics and display system, according toan embodiment;

FIG. 2 is a block diagram illustrating a graphics and displaysub-system, according to an embodiment;

FIG. 3 is an illustration of graphics and display performance controland feedback system 300, according to an embodiment;

FIG. 4 is an illustration of a system for per-frame control ofpresentation time, according to an embodiment;

FIG. 5 is an illustration of a system of per-frame statistics, accordingto an embodiment;

FIG. 6 is an illustration of variable frame rate animation, according toan embodiment;

FIG. 7 illustrates variable frame rate animation for a launch andsuspend operation on an electronic device;

FIG. 8 illustrates variable frame rate animation for an unlock animationon an electronic device;

FIG. 9 is an illustration of variable frame rate coalescing, accordingto an embodiment;

FIG. 10 is a block diagram of a display performance control system,according to an embodiment;

FIG. 11 is a flow diagram illustrating display performance controllogic, according to an embodiment;

FIG. 12 is a flow diagram illustrating display performance control hintlogic, according to an embodiment;

FIG. 13 is a flow diagram illustrating a process of animation frame rateadjustment based on display performance control feedback, according toan embodiment;

FIG. 14 is a flow diagram illustrating display performance controlvariable animation rate logic, according to an embodiment;

FIG. 15 is a block diagram illustrating a multi-layer softwarearchitecture used by a data processing system, according to anembodiment;

FIG. 16 is a block diagram of one embodiment of a computing system;

FIG. 17 is a block diagram of an example network operating environmentfor mobile devices, according to an embodiment; and

FIG. 18 is a block diagram of mobile device architecture, according toan embodiment.

DETAILED DESCRIPTION

Described herein is a system and method to enable improved visualquality provided by higher display refresh rates while balancing theadditional energy requirements of increased refresh rates. Oneembodiment provides a system and method of dynamically controllingdisplay performance based on content to be displayed. One embodimentprovides a hardware accelerated display management framework thatmaintains backwards compatibility with legacy systems that lack displayperformance management hardware acceleration. Embodiments enable smoothdisplayable frame generation and presentation to display devices. Inthis context, smooth means extremely low judder at potentially arbitraryframe arrival intervals.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin conjunction with the embodiment can be included in at least oneembodiment of the invention. The appearances of the phrase “in oneembodiment” in various places in the specification do not necessarilyall refer to the same embodiment. Moreover, specific details ofembodiments are illustrated in detail in certain figures, but indiffering detail in others. One having ordinary skill in the art willunderstand that details shown and/or illustrated for one embodiment maybe combined with details shown and/or illustrated for other, similarembodiments. In certain instances, well-known or conventional detailsare not described in order to provide a concise discussion of variousembodiments of the present invention.

The processes depicted in the figures that follow are performed byprocessing logic that comprises hardware (e.g. circuitry, dedicatedlogic), software (as instructions on a non-transitory machine-readablestorage medium), or a combination of both hardware and software.Although the processes are described below in terms of some sequentialoperations, it should be appreciated that some of the operationsdescribed may be performed in a different order. Moreover, someoperations may be performed in parallel rather than sequentially.

In embodiment described herein, frame generation and display can beenabled via a display architecture comprising multiple components andsub-components across multiple layers of the software and hardware. Inone embodiment the components include a display kernel driver,three-dimensional (3D) graphics application programming interfaces(API), user input frameworks, animation frameworks, window managers,graphics processors, and display engines. Some embodiments additionallyprovide for one or more real-time processors (RTP) configured to executedisplay performance management logic. Additional embodiments provide fora system performance controller that manages and configures performanceparameters for the processing and input/output (I/O) subsystems of anelectronic device

One embodiment enables a system to generate frames at different framedurations without introducing judder. Each frame is associated with aspecified presentation time that is honored by the display system. Adisplay engine can be configured to refresh a display device at anappropriate time to match the animation rate of an incoming frame. Forexample, a 3D game can be configured to generate a frame at logical time0, which represents a current time, or a frame for immediate display.The 3D game can then advance the scene by 8 ms an immediately generate anew frame with a timestamp of 8 ms. The 3D game can then advance thescene by 12 ms and generate a frame with a timestamp of 20 ms. A displaydriver on the system will then see three frames within a display queue,each frame specified for different display times, with a differentamount of time between frames. The frames within a display queue can beprocessed in turn at the specified presentation times to be sent to adisplay device. The display device can then dynamically refresh to matchthe presentation time of each frame.

Embodiments provide a software stack of multiple software elementsconfigured to interface with the hardware components to implement thedisplay system. In one embodiment the software elements include adisplay driver to program a display engine to display image data on thedisplay device, a window server to manage the submission of applicationdata to the display driver, and window clients to generate image datafor display.

The display driver can program the display engine to display image dataon the display device in response to requests to display frames of imagedata. In one embodiment the display driver manages a queue includingconfiguration data for frames to present on the display device. Eachframe has an associated presentation time stamp associated with therequested presentation time for the frame. When the current time matchesa frame's target presentation time, which is the requested presentationtime minus an amount of time to account for the programming of thedisplay hardware, the display driver can program the hardware to displaythe frame. In one embodiment, the image data for a frame can include asingle surface for display or multiple, separate surfaces that arecomposited to create the frame. In one embodiment, the composition isperformed by the display engine when presenting the frame. In oneembodiment, the composition is performed by a graphics engine prior todisplaying the frame.

In addition to a requested presentation time, the configuration data fora frame can include one or more hints from an application, framework, orthe window server. The hints can provide additional details regardingthe frame, such as the relative complexity of the current frame or anupcoming frame in the sequence of frames. For example, a current framecan be composed of several surfaces that are to be blended to create thefinal frame. An upcoming frame can be based on a large number ofgeometric objects or can include a large number of shader programs.These hints can be a leading indicator that additional display orgraphics processing resources may be required. Accordingly, the realtime processor, in one embodiment, can look ahead in the timestamp queueto determine whether the performance controller should be made aware ofa current or pending requirement for additional resources. The real timeprocessor can then signal the performance controller to adjustperformance related parameters for the system. For example, for upcomingcomplex events, the real time processor can pass the hint data for anupcoming frame to the performance controller and, if thermal and/orpower budget is available, the performance controller can increase theclock rate associated with an application processor, graphics processor,and/or display engine. In one embodiment, a communication fabric thatlinks the various hardware components (e.g., application processor,graphics processor, memory, etc.) can also be adjusted based on the hintinformation to temporarily increase fabric bandwidth allocation limitsfor data transfer between the graphics and display components within thesystem.

The window server is configured to master the display of image data fromwindow client applications. An example window server is theCoreAnimation graphics rendering and animation framework of iOS™ and OSX®, which are provided by Apple Inc. of Cupertino Calif. However,embodiments are not limited to any particular operating system, windowserver, or window server implementation. Client applications of thewindow server provide data that describes the content to be displayed ona display device. In one embodiment the clients describe the content viatrees of layers that are composited and rendered into a finished framefor display. The window server can generate implicit animations, inwhich a client application provides images to the window server and thewindow server can generate a sequence of animated frames based on theprovided images. Clients may also enqueue finished frames into an imagequeue of the window server. In one embodiment the window server collectsstatistics generated and provided for a frame (e.g., by the displaydriver in one embodiment) and can vend the statistics to clientapplications as appropriate. For example, for explicit animationsgenerated by client applications, the window server can providestatistics for those frames. For implicit animations, the window servercan monitor the statistics for window server generated frames.

In one embodiment the electronic device includes a system performancecontroller which can be made aware of requested and actual presentationstatistics. If applications, frameworks, or other display clients appearto be struggling to achieve or maintain target animation rate targets,then the performance controller may choose to increase the systemperformance state, provided there is energy and thermal headroomavailable. If the system has been experiencing a high level of energyconsumption over a recent period, then the system performance controllermay choose to lower the system's performance state and request a slowerframe queue drain rate from the display driver. Reducing the systemperformance state may force display clients to become late presentingframes for display, which the clients can respond to by decreasinganimation rates, further lowering energy consumption within the system.

A portion of this disclosure contains material that is subject tocopyright protection. The copyright owner has no objection to thefacsimile reproduction by anyone of the patent document or the patentdisclosure, as it appears in the Patent and Trademark Office patentfiles or records, but otherwise reserves all copyright rightswhatsoever. Copyright 2016 Apple Inc.

Display Architecture

FIG. 1 is a block diagram of a graphics and display system 100,according to an embodiment. The display system provided by embodimentsdescribed herein includes various layers of software and hardware. Inone embodiment, software layers include one or more applications, suchas a user interface (UI) logic 102, a UI client application (UI clientapp 104), a 3D application (3D app 108), and a video application (videoapp 110). Software layers can additionally include applicationframeworks such as a web framework 106 and a UI framework 112. The webframework 106 can provide functionality to web browser applications andcan interface with the UI framework 112. UI client applications 104 anduser interface logic 102 can derive user interface functionality fromthe UI framework 112.

One embodiment provides for a window server 114 and driver logic, suchas display driver and/or graphics driver components of a graphicssub-system 116 and display sub-system 117. The window server 114 canrelay display requests from the user interface logic 102, web framework106, 3D app 108, and/or video app 110. The window server 114 can alsorelay display requests for the UI client app 104 received via the UIframework 112.

One embodiment provides for hardware layers include a graphicssub-system 116 including a graphics processor. The graphics processorcan include one or more graphics engines. Embodiments additionallyprovide for a display sub-system 117 including a display engine.Hardware for the graphics sub-system 116 and display sub-system 117 maybe integrated within a single graphics processor or one or more graphicsprocessors. In one embodiment the single graphics processor or one ormore graphics processors of the graphics sub-system 116 and/or displaysub-system 117 can be integrated into a system on a chip integratedcircuit for use in mobile and/or embedded devices. In one embodiment,hardware for the graphics sub-system 116 and display sub-system 117 maybe in separate devices, for example, where the graphics and displaysystem 100 is configured for use in a desktop or workstation platform.

In one embodiment a performance controller 118 controls power andperformance parameters for the graphics subsystem 116 and the displaysub-system 117. The performance controller can adjust performanceparameters such as a clock frequency or voltage associated with thegraphics subsystem 116 and the display sub-system 117. The voltage,frequency, and/or other power and performance parameters of the graphicssubsystem 116 and/or the display sub-system 117 can be adjusted inunison for each of the graphics subsystem 116 or the display sub-system117, or the power and performance parameters can be independentlyvaried. For example, one or more components of the display sub-system117 can be temporarily power off while the graphics sub-system 116 cancontinue to operate. Likewise, one or more components of the graphicssub-system 116 can be powered off while the display sub-system 117continues to operate.

In one embodiment the performance controller 118 adjusts power andperformance parameters of the graphics subsystem 116 and the displaysub-system 117 based on utilization and workload, such that increasedperformance is available to process larger workloads, within thermallimits imposed by thermal management logic. In one embodiment theperformance controller 118 is a closed loop performance controller(CLPC) and can proactively adjust power and performance parameters ofthe graphics subsystem 116 and the display sub-system 117 based on inputfrom scheduling logic within the graphics sub-system 116.

In one embodiment, a real-time processor 120 (RTP) is additionallyincluded to manage the processing and presentation of frames to and fromthe graphics sub-system 116 and display sub-system 117. The real-timeprocessor 120 can process frame requests from a timestamp queue 121 at arequested display time and present the display request to a displayengine within the display sub-system 117. Entries can be inserted intothe timestamp queue 121 via a driver logic of the graphics sub-system116 and/or the display sub-system 117 at the request of the windowserver 114.

In one embodiment, driver logic associated with the graphics sub-system116 (e.g., graphics driver logic) can be used to insert frames into thetimestamp queue 121 for display. In one embodiment the displaysub-system 117 includes separate software logic to manage the insertionof frames in the timestamp queue 121. In one embodiment, the real-timeprocessor 120 can also provide hints or feedback to the performancecontroller 118 based on contents of the timestamp queue.

The display sub-system 117 can couple with a display device 122.Embodiment provides for a display device 122 has electrical propertiessufficient to support a variable refresh-rate on a frame-by frame basis.For example and in one embodiment, the display device 122 includes adisplay panel having materials sufficient to hold a display charge forlonger than 16 ms without flickering, to enable sub-60 Hz refresh rates.In one embodiment the display device includes refresh logic that canrefresh the display panel at up to 120 Hz.

In one embodiment the real-time processor 120 is configured to monitorthe electrical state of the display device in conjunction withmonitoring the content of the frame and determine the precise durationin which a frame may be displayed without refresh before flickering willbe observable on the display. For example, in some flat panel displaydevices, displaying a luminance value in a range through a paneldepending mid-grey value may be more difficult to display in comparisonto other luminance values and will cause flicker more quickly than otherluminance values. To prevent flickering the real-time processor 120 canissue a command to refresh the panel without software input. Suchautomatic refreshes will not interfere with software requestedpresentation times for new frames. In one embodiment, thenon-interference of automatic refreshes is enforced by the displaysub-system 117 and/or refresh logic within the display device 122.

In one embodiment, applications such as a 3D application (e.g., 3D app108) can provide a completed frame to the window server 114 for display.For example, a client application can generate finished frames using theMetal™ API, OpenGL® API, or any other image API. The 3D application canschedule a frame to be rendered by the graphics-subsystem 116. Thegraphics sub-system 116 can render the frame and the window server 114can facilitate insertion of the completed frame into the timestamp queue121. In one embodiment some applications can provide animationkey-frames to the window server 114. The window server 114 can thengenerate additional, interpolated frames to enable smooth animation ofthe transitions between the key-frames. The key-frames and thetransition frames can be scheduled within the timestamp queue 121 forprocessing.

In one embodiment, applications such as a video application (e.g., videoapp 110) or an application supported by the UI framework 112 can providefinished frames or can provide multiple surfaces to the window server114 to be composited into a final frame. For example, the video app 110can provide a video frame that will be composited with user interfacecontrols or subtitles, or the UI client app 104, via the UI framework112, can request the window server 114 to display a user interface thatis a composite of multiple surfaces.

FIG. 2 is a block diagram illustrating a graphics and display sub-system200, according to an embodiment. The illustrated graphics and displaysub-system 200 provides an additional view of components andsub-components of the graphics and display system 100 of FIG. 1. In oneembodiment the graphics and display sub-system 200 includes driver logic220 (e.g., driver logic 220A, driver logic 220B) to manage the graphicssub-system 116 and/or display sub-system 117. In one embodiment thedriver logic 220 includes graphics driver logic 220A to manage thegraphics sub-system 116 and display driver logic 220B to mange thedisplay sub-system 117.

In one embodiment the graphics sub-system 116 includes a dedicatedphysics engine (phys 208) to use in calculating parameters for futureframes. The physics engine can be used, in one embodiment, to generateprojections for future frames based on current frame data. Theprojections for future frames can be used to pre-render a set of framesfor timed presentation by the display sub-system 117.

One embodiment provides a time-stamp queue 221, which can be a variantof the time-stamp queue 121 of FIG. 1. The illustrated time-stamp queue221 includes an entry 201, 203 for each frame to be processed by thedisplay sub-system 117. In one embodiment each entry 201, 203 in thetime stamp queue 221 includes a submission time 212 in which the frameis inserted into the time-stamp queue 221, frame information 214 thatincludes data that the display sub-system 117 is to use to display aframe. In on embodiment, the frame information 214 for an entry 201, 203includes context and state information for the display sub-system 117 touse to display the frame. The frame information 214 can additionallyinclude one or more memory addresses for surfaces to display as part ofthe frame. In one embodiment the frame information 214 includes arequested presentation time for the frame.

An entry 201, 203 can additionally include and one or more hints 216 tothe display sub-system, which can be provided to the performancecontroller 118 to configure performance parameters for the displaysub-system 117 and/or graphics sub-system 116. In one embodiment, thehint(s) 216 can include an indication as to the complexity of the frameto be displayed, such as a number of surfaces to be composited and/or anindication as to a complexity as to a blend operation to be performed.In one embodiment the hint(s) 216 can include an indication of agraphical complexity of an upcoming scene, such as an indication that ascene includes a larger number of and/or a complex set of shaderoperations to be performed by the graphics sub-system 116. Such hint(s)216 can be provided by the performance controller 118 to enableproactive adjustment of performance related parameters for the graphicssub-system 116 and/or display sub-system 117 based on upcomingworkloads. In one embodiment, the real time processor (RTP 120) cancyclically scan the set of entries in the timestamp queue 221 to readhint(s) 216 from the entries 201, 203 and provide hint relatedinformation to the performance controller 118.

FIG. 3 is an illustration of graphics and display performance controland feedback system 300, according to an embodiment. In one embodimentdisplay system performance can be controlled by processor utilization302 or performance can be controlled by presentation timing 304. In oneembodiment, an application can generate a first frame (Frame i) in whichperformance is controlled by processor utilization (302). A kernel 10mobile framebuffer interrupt can be received via a CADisplayLink objectthat enables an application to synchronize drawing operations to therefresh rate of the display. The application may be an application thatinterfaces with a UI framework (e.g., UIKit) to perform user interfaceoperations. One or more threads of the application can performoperations to generate frame content for display and the content forFrame i can be presented to a display device for immediate display.

In one embodiment, subsequent frames (e.g., Frame i+1) can be generatedand displayed based on a pre-determined presentation timing. A threadoperating on a processor (e.g., CPU) can begin operations to prepare theframe for rendering. A CPU start and end time can be tracked duringframe preparation. During CPU operation for a frame, the CPU can kickrendering operations to a graphics processing unit (GPU 306) and submita swap operation for the frame to the display engine. When the GPUcompletes rendering operations, the display engine can swap the finishedframe with a previously displayed frame.

Statistics associated with the rendering of each frame can be maintainedby the graphics and display performance control and feedback system 300.In one embodiment, a presentation time transaction data structure 308can be maintained, which is used to report presentation timeinformation. Exemplary presentation time transaction informationincludes a human interface device input time (hid_input_time) associatedwith a frame, a CPU start time (cpu_start_time) and CPU finish time(cpu_finish_time) associated with a frame, as well as a GPU start time(gpu_start_time) and GPU finish time (gpu_finish_time). In oneembodiment a requested presentation time (requested_presentation_time)for the frame can be tracked by the presentation time transaction datastructure 308. In one embodiment the presentation time transaction datastructure 308 can also include performance feedback data(performance_feedback), which may be modified by the performancecontroller 118. In one embodiment, an “on glass arrival time” 310 foreach frame is also tracked, which is the time in which a given frame isactually displayed on a display device associated with the displaysub-system. In one embodiment, the set of presentation time statisticscan be provided to clients of the display sub-system, such as the windowserver 114, UI framework 112, 3D app 108, and/or video app 110 as inFIG. 1. In one embodiment, the presentation time statistics can be usedby the clients to adjust rendering operations performed by the clients.In one embodiment the presentation time statistics can also be used bythe performance controller 118 to determine if performance parametersfor the system should be adjusted.

Per-Frame Control of Presentation Time

FIG. 4 is an illustration of a system 400 for per-frame control ofpresentation time, according to an embodiment. In one embodiment, thesystem 400 for per-frame control of frame presentation times 402 canenable variable per-frame presentation durations 404 for a set offrames, in which the display duration for each frame can be determinedat least in part based on the requested presentation time of thesuccessive frame. Each frame can have different presentation durations.For example, frame 411 can be presented at 0 ms and can be presented for8.3 ms. At 8.3 ms, frame 412 can be presented for 12.5 ms. At 20.8 ms,frame 413 can be presented for 16.7 ms. At 37.5 ms, frame 414 can bepresented for 20.8 s until frame 415 is presented. Frame 415 can bepresented for 25 ms.

The duration of each frame can be defined based on a one or more displayquanta (e.g., display quantum 406). The duration of the display quantum406 can be defined based on a display event review cycle of a displaysub-system (e.g., display sub-system 117 of FIG. 1 and FIG. 2). Duringeach display quantum 406, the display sub-system can review a set ofpending display events defined by entries in a timestamp queue (e.g.,entry 201, 203 of timestamp queue 221 of FIG. 2). For example, thedisplay subsystem, in one embodiment, via a real time processor (e.g.,RTP 120 of FIG. 1 and FIG. 2) can operate at a display event review rateof 240 Hz, defining a display quantum 4.166 ms. During each reviewcycle, the display sub-system can determine whether a new frame is to bedisplayed based on a requested display time for the frame. In oneembodiment, when a new frame is not scheduled to be display, the displaysub-system can consult an electrical model of the display device todetermine whether a currently displayed frame should be automaticallyrefreshed. The automatic refresh can be performed without requiring theintervention of the software logic of the display sub-system software,and is configured to not interfere with display sub-system framescheduling.

In one embodiment the display sub-system can specify a duration in whichframe data is to be displayed and the display subsystem will display theframe data on the display during the display quantum 406 in which theframe is to be displayed, assuming the frame is ready for display. Thedisplay sub-system can then either refresh or not refresh the displaydevice at each display quantum 406 based on the electrical properties ofthe display, rather than a pre-defined refresh rate. Thus, in oneembodiment, the refresh rate of the display is at least in partindependent from the frame rate of the content displayed on the display,such that display sub-system can program the display to be updated withnew content at a controlled frame rate while the display device isindependently and automatically refreshed. In one embodiment theautomatic refresh of a frame occurs only when necessary, for example,while displaying a long duration frame and the electrical modelindicates that flicker will occur if the display device is notrefreshed.

Per-Frame Statistics

FIG. 5 is an illustration of a system 500 of per-frame statistics,according to an embodiment. The system 500 of per-frame statistics canmaintain a set of statistics for each frame, for example, using a datastructure similar to the presentation time transaction data structure308 of FIG. 3. Data such as requested presentation times 502 in which aframe is to be displayed, on-glass times 504 in which the frame data isactually displayed, and identifier data for each of the frame 506.Additionally, 60 Hz display times 508 are illustrated for comparison.Using such data, frame statistics (e.g., frame stats 510) can bemaintained for each frame, including times of arrival 512, requestedpresentation 514, and on-glass 516, as well as delta 518 statistics forthe frame. The time of arrival 512, in one embodiment, is the time inwhich a frame is inserted into the timestamp queue (e.g., timestampqueue 221 of FIG. 2). The requested presentation 514 is the time atwhich the frame is requested to be presented on the display device. Theone-glass 516 time for a frame is the time in which the frame isactually displayed on a display device.

The delta 518 statistics reported for a frame differ depending onwhether the time of arrival 512 for the frame is before or after therequested presentation 514 time. If the time of arrival 512 is beforethe requested presentation 514 time, in one embodiment, the displaysub-system can honor the requested presentation time, such that therequested presentation time and the actual presentation (e.g., on-glass)time will be substantially similar. Thus, the delta 518 statistic forthe frame can be recorded as the number of milliseconds in which theframe was early. If the frame arrives late to the timestamp queue, thedelta 518 statistic can be the difference between the requestedpresentation 514 time and the on-glass 516 time, which indicates thedifference between when the frame is displayed and when the frame wasrequested to be displayed.

In embodiment described herein, client applications of a window server(e.g., window server 114 of FIG. 1), can generate images to becomposited or animated. Additionally, client applications can providefinished frames to the window server to be displayed on the displaydevice. Client applications can generate finished frames via a videodecoding API or a 3D rendering API known in the art. A display requestsubmitted by a client application can include a requested presentationtime that the display system will attempt to honor. Video decoders canset a presentation time for a frame based on the content's presentationtime (e.g., encoded frame rate). 3D graphics APIs or other graphicsrendering APIs can set a requested presentation time for each framebased on a determined animation rate for the content. An application canrequest a legacy presentation time based on a default refresh rate orcan specify a specific future timestamp in which a frame will bedisplayed.

After generating a current frame for display, a client application canthen generate several additional frames to be displayed at a specifiedfuture time based on the determined animation rate. For example, aftergenerating a finalized frame based on scene data such as 3D geometrydata, the client can analyze the position, velocity, acceleration, andother object parameters to advance the scene a number of millisecondsinto the future to generate new object data for a future frame, forexample, using a physics engine (e.g., phys 208 of FIG. 2). This futureframe can then be inserted into a window server queue with a requesteddisplay time in the future, before being provided to a displaysub-system timestamp queue (e.g., timestamp queue 221) The generation offuture frames for display at a predetermined time differs from legacyimplementations which perform frame animation at a fixed rate.

If the graphics and display system is operating at a sufficientperformance level, the requested presentation times will be honored. Inone embodiment, the display request for a frame may be submitted beforethe graphics processor has completed rendering the frame. If thecontents of frame are too complex to be completed by the GPU before therequested presentation time, the presentation of the frame may bedelayed. For composite frames, the composition surfaces may be completedby the graphics processor in sufficient time, but the duration of thedisplay engine composition process may cause presentation of a frame tobe delayed. In such scenario, statistics maintained for each frame canbe provided as feedback to software components of the graphics anddisplay system, which can reduce the complexity or animation rates ofsuccessive frames.

For example, if an application or framework is consistently late,animation parameters can be changed to render images for display atlarger interval or reduce the scene complexity in each frame. If theapplication or framework is consistently early, an animation intervalmay be shorted to enable higher temporal visual quality. In oneembodiment, the applications can be configured such that theapplications naturally react to display feedback and reduce thefrequency of display updates when the display system or electronicdevice is under duress, such as during period of high processing load,high thermal load, or low battery energy conditions.

For the exemplary frames of FIG. 5, statistics can be gathered asfollows. Frame 0 can arrive at the timestamp queue at −2 ms, which, forexample, indicates that the frame arrives 2 ms before a timestamp queuereview cycle. Frame 0 can request a presentation time of 0 ms, which inone embodiment indicates a request for a legacy presentation time on thenext 60 Hz boundary. Frame 0 can be displayed on the next 60 Hz boundary(at 0 ms). Thus, frame stats 510 for Frame 0 can be recorded as −2 msarrival 512, 0 ms requested presentation 514 and on-glass 516 times, anda +2 ms delta 518.

Frame stats 510 for Frame 1 can be recorded as 20 ms arrival 512, 12 msrequested presentation 514 time, 24 ms on-glass 516 times, and a −12 msdelta 518. The −12 ms delta is recorded because Frame 1 was late to thetimestamp queue, causing a 12 ms difference between requested and actualdisplay times. A frame can arrive late to the timestamp queue if thegraphical complexity of the frame is such that the GPU and/or CPU cannotcomplete the frame in time to meet the requested display time for theframe. This can indicate that either the CPU and/or GPU should bestepped up to a higher level of performance by the performancecontroller. If the performance level of the CPU and/or GPU cannot beincreased, for example, due to thermal or energy limits imposed on thesystem, late frames can indicate that the animation frame rate orgraphical complexity of the frames should be reduced.

Frame stats 510 for Frame 2 can be recorded as 38 ms arrival 512, 42 msrequested presentation 514 and on-glass 516 times, with a +4 ms delta518. Either due to reduced complexity relative to frame 1, or a reducedanimation rate, Frame 2 arrived 4 ms before its requested presentationtime, allowing the requested presentation time to be honored by thegraphics display and display system. Frame 2 is displayed forapproximately 91 ms, during which the display device can beautomatically and adaptively refreshed as necessary based on thecontents of Frame 2 and the electrical properties of the display.

Frame stats 510 for Frame 3 can be recorded as 130 ms arrival 512, 82 msrequested presentation 514 time, 133 ms on-glass 516 times, and a −51 msdelta 518. The −51 ms delta is recorded because Frame 1 was late to thetimestamp queue, causing a 51 ms difference between requested and actualdisplay times. In one embodiment, a large difference between a requestedpresentation time for a frame and the arrival time of a frame in thetimestamp queue can indicate that the content of the frame issignificantly more complex than can be smoothly displayed at the currentperformance level of the graphics and display system. Additionally, suchlarge difference may be the result of the performance controller for thesystem reducing the performance level of the system due to recent highlevels of energy consumption, which can cause rendered frames to be lateand trigger a reduction in frame complexity at the software level,further reducing system energy consumption.

Variable Frame Rate Animation

The graphics and display system provided by embodiments described hereincan be configured to enable variable frame rate animation withinapplications. An application can present a sequence of frames having ananimation frame rate that varies based on the content displayed. Basedon the variable animation frame rate of the content, the display devicecan have a variable refresh rate that is based on the animation framerate provided by the application.

FIG. 6 is an illustration of variable frame rate animation 600,according to an embodiment. In one embodiment an application, such as ane-mail application, can provide content 601 for display. The applicationcan perform a scroll 602 operation for the displayed content 601 basedon input (e.g., touchpad, touchscreen, scroll wheel, etc.) received froma graphical interface. In response to an initial input to scroll thedisplayed data, the application can generate a sequence of successiveframes at an initial animation frame rate 604 based on the velocity 606or intensity of the initial scrolling input. In one embodiment displaylogic can compute an animation rate curve 607 for the scroll 602operation based on animation logic. The animation rate curve 607 canthen be quantized to specific frame rates 608A-F at which animationframes for the scroll operation can be generated.

For a fast initial input, such as a fast touch or scroll gesture, theapplication can generate a sequence of frames having a high animationframe rate 604 (e.g., frame rate 608A, frame rate 608B, frame rate608C). The scroll rate can be slowed over time (e.g., frame rate 608D,frame rate 608E, frame rate 608F), for example in response to a fade ofthe initial input, which can reduce the amount of power consumed by thedisplay system.

The display device can be refreshed at refresh rate matching theapplication animation frame rate to increase the clarity of the sequenceof frames. In other words, instead of animating frames based on theframe rate of the display, the display is refreshed based on theanimation rate of the content. Accordingly, an initially high animationrate can result in an initially high display refresh rate (e.g., 120Hz), which results in an increase in display power consumption relativeto a 60 Hz refresh rate, but produces enhanced visual quality outputrelative to 60 Hz. The initially high animation rate can be reduced overtime, resulting in power savings relative to 120 Hz (e.g., frame rates608B-608C), then power savings relative to 60 Hz (e.g., frame rates608D-608F).

While FIG. 6 provides scrolling as an example of variable frame rateanimation, animation rate curves can be determined for other displayanimations, such as animations associated with application launch andsuspend, device unlock, home screen scrolling, and pushing and poppingdisplay views. Adaptive refresh can be performed for each of theseanimations, such that the refresh rate of the display conforms to theanimation rate for the frames generated for the animation. FIG. 7-10illustrate additional animation/refresh rate curves for exemplaryoperations.

FIG. 7 illustrates variable frame rate animation 700 for a launch andsuspend operation on an electronic device. The animation for launch andsuspend includes a launch portion in which an animation for anapplication launch is displayed and a suspend portion in which ananimation for application suspend is displayed. During the launchanimation, a first set of frames can be generated at a first animationrate 702A, such as 120 Hz. The animation can begin at an initially highframe rate relative to successive frames due to the nature of theanimation displayed, such that higher frame rates during the initialperiod of the launch animation provides improved visual impact and/orvisual quality relative to animations at a lower frame rate.

After a period of time at the first animation rate 702A, the animationrate can be reduced to a second animation rate 702B that issignificantly lower than the first animation rate (e.g., 60 Hz). In oneembodiment the animation rate of the launch animation can be morequickly stepped down to a third animation rate 702C (e.g., 40 Hz) and afourth animation rate (e.g., 24 Hz). The display system, for eachanimation rate 702A-702D, can base a requested refresh rate for thedisplay on the animation rates for the content.

In one embodiment, animation rates for a suspend animation can bedefined similarly, with a high initial animation/frame rate 704Afollowed by a tapering of the frame rate to a second animation rate 704Cand a third animation rate 704D.

FIG. 8 illustrates variable frame rate animation 800 for an unlockanimation on an electronic device. The unlock animation can be displayduring, or in response to an input, for example, a touch screen input,to transition the electronic device from a locked to an unlocked state.The animation can begin at first animation rate 802A, which is a lowanimation frame rate relative to successive frame rates. A frame rateramp can occur in which successively higher animation rates (animationrate 802B, animation rate 802C, animation rate 802D) are stepped throughuntil a maximum animation rate 802E (e.g., 120 Hz) is reached. Themaximum animation rate 802E can be maintained for a period of timebefore an animation rate taper occurs, where the animation rate issuccessively reduced (animation rate 802F, animation rate 802G,animation rate 802H).

In general, variable frame rate animation provided by embodiments can beclassified into multiple stages. First, an initial frame rate rampoccurs, which may be immediate as in the launch and suspend animation ofFIG. 7, or more gradual as with the unlock animation of FIG. 8. Second,a high animation rate is maintained for a period. Third, the highanimation rate is tapered to a lower animation rate. By following suchrefresh rate patterns, embodiments can approach the visual quality of ahigh sustained animation rate while realizing display system powersavings, for example, as illustrated in FIG. 6. The multi-stage framerate control can be applied to various animations in addition to thoseillustrated herein, such as a home screen scrolling and pushing andpopping user interface views. In each instance, the refresh rate of adisplay can be synchronized with the animation rates, such that thedisplay is rapidly refreshed during periods of fast animation and slowlyrefreshed during periods of slow animation.

In one embodiment, variable animation rates and animation rate baseddisplay refreshing can be used to coalesce the animation and frame ratesof multiple applications. For example, one embodiment provides for atablet computing device in which multiple foreground applications may bedisplayed simultaneously and each of the multiple foregroundapplications can display animated content on the screen. Eachapplication can have different animation frame rate and associatedpresentation times, such that the separate applications are not tied tothe same animation rate. However, it may be advantageous to at leastpartially synchronize the animation rates of the applications.

In one embodiment the display system is configured to attempt tosynchronize the applications to a common phase to enable a displayingcontent for both applications at a lower refresh rate. For example, twoapplications generating 30 Hz animations can be configured to generateframes at the same time to enable the display device to refresh at 30Hz. Additionally, the presentation times for the applications can becoalesced such that, if allowed by the application and the frames inquestion are available for display, the display times for one or moreapplications may be adjusted to enable the frames of the multipleapplications to be presented in the same refresh cycle. However, if thedisplay system is unable to keep the animation frame rates of theapplications in phase, the display device can be refreshed at a higherrefresh rate (e.g., 60 Hz) to honor the requested presentation times forthe applications. As an additional example, if two applications havecontent animated at 60 Hz and the content cannot be phase aligned, thedisplay device can be refreshed at 120 Hz to honor the requestedpresentation times for the applications.

FIG. 9 is an illustration of variable frame rate coalescing 900,according to an embodiment. In one embodiment a first application 910and a second application 920 can be displayed on a display device,wherein each application renders frames at independent target animationrates. In an exemplary condition in which a frame (e.g., frame 912) ofthe first application 910 is out of phase with a frame (e.g., frame 922)of the second application 920, the swap time can be adjusted 902 (e.g.,delayed) to align the frames, such that, for example, frame 912 andframe 922 are each scheduled to be swapped during the same displayquantum, where frame 912 is replaced with frame 913 and frame 922 isreplaced with frame 923. In addition to adjusting a requestedpresentation time of a frame to delay frame presentation, one embodimentprovides logic that can advance 904 a scheduled presentation time forframe, such that frame 914 for the first application 910 can bepresented during same display quantum as frame 924 of the secondapplication 920.

FIG. 10 is a block diagram of a display performance control system 1000,according to an embodiment. The illustrated modules are an exemplaryrepresentation of software or hardware logic of the display performancecontrol system 1000. In one embodiment the display performance controlsystem 1000 includes a display performance control module 1010. Thedisplay performance control module 1010 can include a display requestinterface module 1001, a display event module 1002, a displayprogramming module 1003, a display hint module 1004, a performancecontroller interface module 1005, a refresh rate adjustment module 1006,a display statistics module 1007, and window manager interface module1008, and/or an animation rate adjustment module 1009. The illustratedmodules are an exemplary representation of software or hardware logic ofthe display performance control system 1000.

In one embodiment the display request interface module 1001 isconfigured to receive a request to display one or more frames of data.The request can include a requested presentation time in which the oneor more frames of data are to be displayed. The one or more frames canbe generated by a graphics processor described herein. The display eventmodule 1002 can be configured to generate one or more display eventsassociated with the one or more frames of data. The display programmingmodule 1003 can be configured to program the display engine based on theone or more display events.

In one embodiment, one or more hints can be included in the generateddisplay events by the display hint module 1004. The display hint module1004 can provide hints to the display system based on a complexity ofcurrent or upcoming frames. In one embodiment the display hint module1004 can process a first hint included in a display request and, basedon the first hint, include a second hint in the display event. Theformats of the first hint and the second hint can be the same or can bedifferent, such that the first hint is inserted into the display eventor the second hint can be derived based on the first hint. In oneembodiment the second hint within the display event can be provided to aperformance controller via a performance controller interface module1005. The performance controller can be configured to adjust one or moreperformance related parameters for the display system based on thesecond hint.

In one embodiment the display performance control system 1000, inresponse to a display event, can enable the refresh rate adjustmentmodule 1006 to adjust a refresh rate of a display based on a processeddisplay event. The refresh rate adjustment module 1006 can dynamicallyadjust the refresh rate of the display device based on the contentdisplayed or to be displayed on the display device.

In one embodiment the display performance control system 1000 canconfigure the display statistics module 1007 to maintain per-framestatistics of frames that are displayed on the display device. Thedisplay statistics module 1007 can manage a set of statistics associatedwith a display request received via the display request interface module1001. The display statistics module 1007 can be further configured toperiodically submit the set of statistics to a window manager of thedisplay system via a window manager interface module 1008. The windowmanager, after receiving the statistics, can enable an animation rateadjustment module 1009 to adjust an animation rate of animationsgenerated by the window manager or request an animation rate adjustmentfrom client applications of the window manager.

Display Performance Control Logic

FIGS. 11-14 illustrate logic operations to perform the displayperformance control processes described herein. The illustrated logicoperations can be performed by various components of the graphics anddisplay system 100 of FIG. 1 and/or the graphics and display sub-system200 as in FIG. 2 using the techniques and data structures hereindescribed throughout the present specification.

FIG. 11 is a flow diagram illustrating display performance control logic1100, according to an embodiment. In one embodiment the displayperformance control logic 1100 can receive a request to display one ormore frames of data on a display device, as shown at 1102. The one ormore frames of data can be generated via one or more processorsdescribed herein. In one embodiment the request includes a requestedpresentation time in which the one or more frames of data are to bedisplayed.

The display performance control logic 1100 can further derive a targetpresentation time based on the requested presentation time, as shown at1104. The target presentation time can be derived based on a programmingtime for a display engine. In other words, the target presentation timeis an adjustment of the requested presentation time to account for theamount of time required to program the display engine with theinformation used to display the frame.

The display performance control logic 1100 can further program thedisplay engine to present the one or more frames of data to the displaydevice for display, as shown at 1106. The display performance controllogic 1100 can then cause the display engine to adjust a refresh rate ofthe display device based on the target presentation time of the one ormore frames of data, as shown at 1108.

FIG. 12 is a flow diagram illustrating display performance control hintlogic 1200, according to an embodiment. The performance control hintlogic 1200, in one embodiment, can receive a request to display one ormore frames of data, as shown at 1202. The request can include a firsthint to indicate a complexity of the one or more frames of data. Theperformance control hint logic 1200 can then generate one or moredisplay events based on the request, as shown at 1204. The one or moredisplay events can include a second hint that can be derived from thefirst hint. The performance control hint logic 1200 can then signal aperformance controller associated with the one or more processors toadjust one or more performance related parameters of the one or moreprocessors based on the second hint, as shown at 1206.

FIG. 13 is a flow diagram illustrating a process 1300 of animation framerate adjustment based on display performance control feedback, accordingto an embodiment. The illustrated process 1300 of animation frame rateadjustment can be performed based on the generation of one of moreframes of data at a first frame rate, as shown at 1302. For example, agraphic application can generate frames for display and request a windowmanager to submit a request to display logic to display the one or moreframes of data, as shown at 1304. Display logic, at 1306, can receivethe request and generate one or more display events for a display enginebased on the request.

The display engine can process the display events to display the one ormore frames of data, as shown at 1308. In one embodiment a real timeprocessor can be configured to provide display events to the displayengine. The display events can be provided, for example, in the form oftimestamp queue entries (e.g., entry 201, 203 of the timestamp queue 221of FIG. 2).

In one embodiment, the display engine can generate a set of statisticsassociated with processing the display events, as shown at 1310. The setof statistics can be used to adjust the first frame rate to a secondframe rate based on the set of statistic, as shown at 1312.

FIG. 14 is a flow diagram illustrating display performance controlvariable animation rate logic 1400, according to an embodiment. Thedisplay performance control variable animation rate logic 1400 canreceive input to interact with an image presented via the display, asshown at 1402. The display performance control variable animation ratelogic 1400 can then generate a sequence of frames in response to theinput, the sequence of frames having an animation frame rate independentof a current refresh rate of the display device, as shown at 1404. Theanimation frame rate can be determined based on a parameter associatedwith the input.

In one embodiment the display performance control variable animationrate logic 1400 can configure display events associated with thesequence of frames, where the display events include configuration datato configure a display engine to display one or more frames in thesequence of frames, as shown at 1406. Each frame in the sequence offrames can have a requested presentation time based on the animationframe rate.

In one embodiment the display performance control variable animationrate logic 1400 can process a first display event associated with afirst frame in the sequence of frames, as shown at 1408. In oneembodiment processing the first display event includes readingconfiguration data to configure the display engine to display the firstframe.

In one embodiment the display performance control variable animationrate logic 1400 can then present the configuration data to the displayengine at a display time based on the requested presentation time, asshown at 1410.

In one embodiment the display performance control variable animationrate logic 1400 can then dynamically refresh the display device based onthe display time to display the sequence of frames, as shown at 1412.

FIG. 15 is a block diagram illustrating a multi-layer softwarearchitecture 1500 used by a data processing system, according to anembodiment. The software components are illustrated with a divisionbetween user space and a kernel space. Although other arrangements arepossible, user applications (e.g., user application 1502), and someoperating system components (e.g., operating system user interface layer1506, and the core operating system layer 1510) execute in user space.In kernel space, the operating system kernel and a set of device driversoperate in the kernel and device driver layer 1512. The kernel anddevice driver layer 1512 manage the underlying functionality of theoverall operating system and provide a formalized and secure mechanismfor user space software to access data processing system hardware.

A user interface (UI) application framework 1504 provides a mechanismfor the user application 1502 to access UI services provided by theoperating system (OS) UI layer 1506. Underlying operating systemfunctions that are not related to the user interface can be performed inthe core operating system layer 1510. One or more data managementframeworks, such as a core app framework 1508 can be made available to auser application to facilitate access to operating system functions.

The exemplary user application 1502 may be any one of a plurality ofuser applications. Each user application 1502 can include one or moreprocesses or tasks, which may spawn multiple threads. The userapplication 1502 can access instructions in an exemplary UI appframework 1504 for creating and drawing graphical user interface objectssuch as icons, buttons, windows, dialogs, controls, menus, and otheruser interface elements. The UI application framework 1504 also providesadditional functionality including menu management, window management,and document management, as well as file open and save dialogs,drag-and-drop, and copy-and-paste handling.

The core operating system layer 1510 contains operating systemcomponents that implement features including and related to applicationsecurity, system configuration, graphics and media hardwareacceleration, and directory services. Multiple application frameworks,including the core app framework 1508, provide a set of APIs to enable auser application 1502 to access core services that are essential to theapplication, but are not directly related to the user interface of theapplication. The core app framework 1508 can facilitate an application'saccess to database services, credential and security services, backupservices, data synchronization services, and other underlyingfunctionality that may be useful to an application.

The core app framework 1508, or equivalent application frameworks, canprovide access to remote server based storage for functionalityincluding synchronized document storage, key-value storage, and databaseservices. Key-value storage allows a user application 1502 to sharesmall amounts of data such as user preferences or bookmarks amongmultiple instances of the user application 1502 across multiple clientdevices. The user application 1502 can also access server-based,multi-device database solutions via the core app framework 1508.

The systems and methods described herein can be implemented in a varietyof different data processing systems and devices, includinggeneral-purpose computer systems, special purpose computer systems, or ahybrid of general purpose and special purpose computer systems.Exemplary data processing systems that can use any one of the methodsdescribed herein include desktop computers, laptop computers, tabletcomputers, smart phones, cellular telephones, personal digitalassistants (PDAs), embedded electronic devices, or consumer electronicdevices.

FIG. 16 is a block diagram of one embodiment of a computing system 1600.The computing system illustrated in FIG. 16 is intended to represent arange of computing systems (either wired or wireless) including, forexample, desktop computer systems, laptop computer systems, tabletcomputer systems, cellular telephones, personal digital assistants(PDAs) including cellular-enabled PDAs, set top boxes, entertainmentsystems or other consumer electronic devices. Alternative computingsystems may include more, fewer and/or different components. Thecomputing system of FIG. 16 may be used to provide the computing deviceand/or the server device.

Computing system 1600 includes bus 1635 or other communication device tocommunicate information, and processor 1610 coupled to bus 1635 that mayprocess information. While computing system 1600 is illustrated with asingle processor, computing system 1600 may include multiple processorsand/or co-processors 1610. Computing system 1600 further may includerandom access memory (RAM) or other dynamic storage device 1620(referred to as main memory), coupled to bus 1635 and may storeinformation and instructions that may be executed by processor(s) 1610.Main memory 1620 may also be used to store temporary variables or otherintermediate information during execution of instructions by processor1610.

Computing system 1600 may also include read only memory (ROM) 1630and/or another storage device 1640 coupled to bus 1635 that may storeinformation and instructions for processor(s) 1610. Data storage device1640 may be coupled to bus 1635 to store information and instructions.Data storage device 1640 such as flash memory or a magnetic disk oroptical disc and corresponding drive may be coupled to computing system1600.

Computing system 1600 may also be coupled via bus 1635 to display device1650, such as a cathode ray tube (CRT), a liquid crystal display (LCD),or a light emitting diode (LED) display, to display information to auser. Computing system 1600 can also include an alphanumeric inputdevice 1660, including alphanumeric and other keys, which may be coupledto bus 1635 to communicate information and command selections toprocessor(s) 1610. Another type of user input device is cursor control1670, such as a touchpad, a mouse, a trackball, or cursor direction keysto communicate direction information and command selections toprocessor(s) 1610 and to control cursor movement on display 1650.Computing system 1600 may also receive user input from a remote devicethat is communicatively coupled to computing system 1600 via one or morenetwork interfaces 1680.

Computing system 1600 further may include one or more networkinterface(s) 1680 to provide access to a network, such as a local areanetwork. Network interface(s) 1680 may include, for example, a wirelessnetwork interface having antenna 1685, which may represent one or moreantenna(e). Computing system 1600 can include multiple wireless networkinterfaces such as a combination of WiFi, Bluetooth®, near fieldcommunication (NFC), and/or cellular telephony interfaces. Networkinterface(s) 1680 may also include, for example, a wired networkinterface to communicate with remote devices via network cable 1687,which may be, for example, an Ethernet cable, a coaxial cable, a fiberoptic cable, a serial cable, or a parallel cable.

In one embodiment, network interface(s) 1680 may provide access to alocal area network, for example, by conforming to IEEE 802.11 b and/orIEEE 802.11 g standards, and/or the wireless network interface mayprovide access to a personal area network, for example, by conforming toBluetooth standards. Other wireless network interfaces and/or protocolscan also be supported. In addition to, or instead of, communication viawireless LAN standards, network interface(s) 1680 may provide wirelesscommunications using, for example, Time Division, Multiple Access (TDMA)protocols, Global System for Mobile Communications (GSM) protocols, CodeDivision, Multiple Access (CDMA) protocols, Long Term Evolution (LTE)protocols, and/or any other type of wireless communications protocol.

Computing system 1600 can further include one or more energy sources1605 and an one or more energy measurement systems 1645. Energy sources1605 can include an AC/DC adapter coupled to an external power source,one or more batteries, one or more charge storage devices, a USBcharger, or other energy source. Energy measurement systems include atleast one voltage or amperage measuring device that can measure energyconsumed by the computing device 1600 during a predetermined period oftime. Additionally, one or more energy measurement systems can beincluded that measure, e.g., energy consumed by a display device,cooling subsystem, WiFi subsystem, or other frequently-used or highconsumption subsystem.

FIG. 17 is a block diagram of an example network operating environment1700 for mobile devices, according to an embodiment. Mobile device 1702Aand mobile device 1702B can, for example, communicate 1720 over one ormore wired and/or wireless networks 1710 to perform data communication.For example, a wireless network 1712, e.g., a cellular network, cancommunicate with a wide area network (WAN) 1714, such as the Internet,by use of a gateway 1716. Likewise, an access device 1718, such as amobile hotspot wireless access device, can provide communication accessto the wide area network 1714.

In some implementations, both voice and data communications can beestablished over the wireless network 1712 and/or the access device1718. For example, mobile device 1702A can place and receive phone calls(e.g., using VoIP protocols), send and receive e-mail messages (e.g.,using POP3 protocol), and retrieve electronic documents and/or streams,such as web pages, photographs, and videos, over the wireless network1712, gateway 1716, and wide area network 1714 (e.g., using TCP/IP orUDP protocols). In some implementations, mobile device 1702A can placeand receive phone calls, send and receive e-mail messages, and retrieveelectronic documents over the access device 1718 and the wide areanetwork 1714. In some implementations, mobile device 1702A or mobiledevice 1702B can be physically connected to the access device 1718 usingone or more cables and the access device 1718 can be a personalcomputer. In this configuration, mobile device 1702A or mobile device1702B can be referred to as a “tethered” device.

Mobile device 1702A or mobile device 1702B can communicate with one ormore services, such as a navigation service 1730, a messaging service1740, a media service 1750, a storage service 1760, and a telephonyservice 1770 over the one or more wired and/or wireless networks 1710.For example, the navigation service 1730 can provide navigationinformation, e.g., map information, location information, routeinformation, and other information. The messaging service 1740 can, forexample, provide e-mail and/or other messaging services. The mediaservice 1750 can, for example, provide access to media files, such assong files, audio books, movie files, video clips, and other media data.A storage service 1760 can provide network storage capabilities tomobile device 1702A and mobile device 1702B to store documents and mediafiles. A telephony service 1770 can enable telephonic communicationbetween mobile device 1702A and mobile device 1702B, or between a mobiledevice and a wired telephonic device. The telephony service 1770 canroute voice over IP (VoIP) calls over the wide area network 1714 or canaccess a cellular voice network (e.g., wireless network 1712). Otherservices can also be provided, including a software update service toupdate operating system software or client software on the mobiledevices.

Mobile device 1702A or 1702B can also access other data and content overthe one or more wired and/or wireless networks 1710. For example,content publishers, such as news sites, RSS feeds, web sites, blogs,social networking sites, developer networks, etc., can be accessed via aweb browser as described herein. For example, mobile device 1702A and/ormobile device 1702B can execute browser software to access web sitesprovided by servers accessible via the wide area network 1714.

FIG. 18 is a block diagram of mobile device architecture 1800, accordingto an embodiment. The mobile device architecture 1800 includes a includea memory interface 1802, a processing system 1804 including one or moredata processors, image processors and/or graphics processing units, anda peripherals interface 1806. The various components can be coupled byone or more communication buses or signal lines. The various componentscan be separate logical components or devices or can be integrated inone or more integrated circuits, such as in a system on a chipintegrated circuit.

The memory interface 1802 can be coupled to memory 1850, which caninclude high-speed random access memory such as static random accessmemory (SRAM) or dynamic random access memory (DRAM) and/or non-volatilememory, such as but not limited to flash memory (e.g., NAND flash, NORflash, etc.).

Sensors, devices, and subsystems can be coupled to the peripheralsinterface 1806 to facilitate multiple functionalities. For example, amotion sensor 1810, a light sensor 1812, and a proximity sensor 1814 canbe coupled to the peripherals interface 1806 to facilitate the mobiledevice functionality. Other sensors 1816 can also be connected to theperipherals interface 1806, such as a positioning system (e.g., GPSreceiver), a temperature sensor, a biometric sensor, or other sensingdevice, to facilitate related functionalities. A camera subsystem 1820and an optical sensor 1822, e.g., a charged coupled device (CCD) or acomplementary metal-oxide semiconductor (CMOS) optical sensor, can beutilized to facilitate camera functions, such as recording photographsand video clips.

Communication functions can be facilitated through one or more wirelesscommunication subsystems 1824, which can include radio frequencyreceivers and transmitters and/or optical (e.g., infrared) receivers andtransmitters. The specific design and implementation of thecommunication subsystems 1824 can depend on the communication network(s)over which a mobile device is intended to operate. For example, a mobiledevice including the illustrated mobile device architecture 1800 caninclude wireless communication subsystems 1824 designed to operate overa GSM network, a CDMA network, an LTE network, a Wi-Fi network, aBluetooth network, or any other wireless network. In particular, thewireless communication subsystems 1824 can provide a communicationsmechanism over which a client browser application can retrieve resourcesfrom a remote web server.

An audio subsystem 1826 can be coupled to a speaker 1828 and amicrophone 1830 to facilitate voice-enabled functions, such as voicerecognition, voice replication, digital recording, and telephonyfunctions.

The I/O subsystem 1840 can include a touch screen controller 1842 and/orother input controller(s) 1844. The touch-screen controller 1842 can becoupled to a touch sensitive display system 1846. The touch sensitivedisplay system 1846 and touch screen controller 1842 can, for example,detect contact and movement and/or pressure using any of a plurality oftouch and pressure sensing technologies, including but not limited tocapacitive, resistive, infrared, and surface acoustic wave technologies,as well as other proximity sensor arrays or other elements fordetermining one or more points of contact with a touch sensitive displaysystem 1846. Display output for the touch sensitive display system 1846can be generated by a display controller 1843. In one embodiment thedisplay controller 1643 can provide frame data to the touch sensitivedisplay system 1846 at a variable frame rate.

In one embodiment a sensor controller 1844 is included to monitor,control, and/or processes data received from one or more of the motionsensor 1810, light sensor 1812, proximity sensor 1814, or other sensors1816. The sensor controller 1844 can include logic to interpret sensordata to determine the occurrence of one of more motion events oractivities by analysis of the sensor data from the sensors.

In one embodiment the I/O subsystem 1840 includes other inputcontroller(s) 1845 that can be coupled to other input/control devices1848, such as one or more buttons, rocker switches, thumb-wheel,infrared port, USB port, and/or a pointer device such as a stylus, orcontrol devices such as an up/down button for volume control of thespeaker 1828 and/or the microphone 1830.

In one embodiment, the memory 1850 coupled to the memory interface 1802can store instructions for an operating system 1852, including portableoperating system interface (POSIX) compliant and non-compliant operatingsystem or an embedded operating system. The operating system 1852 mayinclude instructions for handling basic system services and forperforming hardware dependent tasks. In some implementations, theoperating system 1852 can be a kernel.

The memory 1850 can also store communication instructions 1854 tofacilitate communicating with one or more additional devices, one ormore computers and/or one or more servers, for example, to retrieve webresources from remote web servers. The memory 1850 can also include userinterface instructions 1856, including graphical user interfaceinstructions to facilitate graphic user interface processing.

Additionally, the memory 1850 can store sensor processing instructions1858 to facilitate sensor-related processing and functions; telephonyinstructions 1860 to facilitate telephone-related processes andfunctions; messaging instructions 1862 to facilitateelectronic-messaging related processes and functions; web browserinstructions 1864 to facilitate web browsing-related processes andfunctions; media processing instructions 1866 to facilitate mediaprocessing-related processes and functions; location servicesinstructions including GPS and/or navigation instructions 1868 and Wi-Fibased location instructions to facilitate location based functionality;camera instructions 1870 to facilitate camera-related processes andfunctions; and/or other software instructions 1872 to facilitate otherprocesses and functions, e.g., security processes and functions, andprocesses and functions related to the systems. The memory 1850 may alsostore other software instructions such as web video instructions tofacilitate web video-related processes and functions; and/or webshopping instructions to facilitate web shopping-related processes andfunctions. In some implementations, the media processing instructions1866 are divided into audio processing instructions and video processinginstructions to facilitate audio processing-related processes andfunctions and video processing-related processes and functions,respectively. A mobile equipment identifier, such as an InternationalMobile Equipment Identity (IMEI) 1874 or a similar hardware identifiercan also be stored in memory 1850.

Each of the above identified instructions and applications cancorrespond to a set of instructions for performing one or more functionsdescribed above. These instructions need not be implemented as separatesoftware programs, procedures, or modules. The memory 1850 can includeadditional instructions or fewer instructions. Furthermore, variousfunctions may be implemented in hardware and/or in software, includingin one or more signal processing and/or application specific integratedcircuits.

In the foregoing specification, the invention has been described withreference to specific embodiments thereof. It will, however, be evidentthat various modifications and changes can be made thereto withoutdeparting from the broader spirit and scope of the invention. Thespecification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense.

What is claimed is:
 1. A method of generating and displaying data on adisplay device, the method comprising: receiving an input to interactwith an image presented via the display; generating a sequence of framesin response to the input, the sequence of frames having an animationframe rate independent of a current refresh rate of the display device,the animation frame rate determined based on a parameter associated withthe input; configuring display events associated with the sequence offrames, the display events including configuration data to configure adisplay engine to display one or more frames in the sequence of frames,each frame in the sequence of frames having a requested presentationtime based on the animation frame rate; processing a first display eventassociated with a first frame in the sequence of frames, whereinprocessing the first display event includes reading the configurationdata to configure the display engine to display the first frame;presenting the configuration data to the display engine at a displaytime based on the requested presentation time; and dynamicallyrefreshing the display device based on the display time to display thesequence of frames.
 2. The method as in claim 1, additionally comprisingsuccessively refreshing the display device at the animation frame rateof the sequence of frames.
 3. The method as in claim 1, wherein theanimation frame rate is a first animation frame rate and the methodadditionally comprises: displaying the sequence of frames at the firstanimation frame rate for a period; adjusting the animation frame rate toa second animation frame rate after the period; and dynamicallyrefreshing the display device at the second animation frame rate.
 4. Themethod as in claim 1, wherein the input is received from a touch inputdevice and the parameter associated with the input is a velocity orpressure associated with the input.
 5. The method as in claim 1, whereinconfiguring display events associated with the sequence of framesincludes storing the display events into a data structure in memory. 6.The method as in claim 1, additionally comprising: periodicallyreviewing the display events at a first rate to determine whether toprocess a display event, wherein each review cycle of the display eventsdefines a display quantum; processing a display event when the displayevent is to occur during a current display quantum; and dynamicallyrefreshing the display device at a second rate to display a frameassociated with the display event, the first rate different from thesecond rate.
 7. The method as in claim 6, additionally comprising:displaying the first frame for a first number of display quanta;displaying a second frame for a second number of display quanta; and notrefreshing the display between the first frame and the second frame. 8.A method of determining a refresh rate for a display device, the methodcomprising: receiving a first request to display a first sequence ofimages associated with a first application, the first sequence of imageshaving a first animation frame rate; receiving a second request todisplay a second sequence of images associated with a secondapplication, the second sequence of images having a second animationframe rate; determining a common refresh interval of the first animationframe rate and the second animation frame rate; and refreshing thedisplay device at a refresh rate based on the common refresh interval.9. The method as in claim 8, additionally comprising coalescing thefirst animation frame rate and the second animation frame rate into acoalesced animation frame rate, and deriving the refresh rate for thedisplay device based on the coalesced animation frame rate.
 10. Themethod as in claim 9, wherein the coalesced animation frame rate isequal to the first animation frame rate and a multiple of the secondanimation frame rate.
 11. The method as in claim 8, wherein the commonrefresh interval is based on one or more presentation quanta and aminimum refresh interval is a multiple of a presentation quantum.
 12. Anon-transitory machine readable medium storing instructions which, whenexecuted by one or more processors, cause the one or more processors toperform operations comprising: receiving an input to interact with animage presented via a display device; generating a sequence of frames inresponse to the input, the sequence of frames having an animation framerate independent of a current refresh rate of the display device, theanimation frame rate determined based on a parameter associated with theinput; configuring display events associated with the sequence offrames, the display events including configuration data to configure adisplay engine to display one or more frames in the sequence of frames,each frame in the sequence of frames having a requested presentationtime based on the animation frame rate; processing a first display eventassociated with a first frame in the sequence of frames, the processingincluding reading the configuration data to configure the display engineto display the first frame; presenting the configuration data to thedisplay engine at a display time based on the requested presentationtime; and dynamically refreshing the display device based on the displaytime to display the sequence of frames.
 13. The non-transitory machinereadable medium as in claim 12, the operations additionally comprisingsuccessively refreshing the display device at the animation frame rateof the sequence of frames.
 14. The non-transitory machine readablemedium as in claim 12, wherein the animation frame rate is a firstanimation frame rate and the operations additionally comprise:displaying the sequence of frames at the first animation frame rate fora period; adjusting the animation frame rate to a second animation framerate after the period; and dynamically refreshing the display device atthe second animation frame rate.
 15. The non-transitory machine readablemedium as in claim 12, wherein the input is received from a touch inputdevice and the parameter associated with the input is a velocity orpressure associated with the input.
 16. The non-transitory machinereadable medium as in claim 12, the operations additionally comprising:periodically reviewing the display events at a first rate to determinewhether to process a display event, wherein each review cycle of thedisplay events defines a display quantum; processing a display eventwhen the display event is to occur during a current display quantum; anddynamically refreshing the display device at a second rate to display aframe associated with the display event, the first rate different fromthe second rate.
 17. The non-transitory machine readable medium as inclaim 16, the operations additionally comprising: displaying the firstframe for a first number of display quanta; displaying a second framefor a second number of display quanta; and not refreshing the displaybetween the first frame and the second frame.
 18. A display systemcomprising: a display device to display a first frame including a firstimage; an input device to receive a sequence of inputs to interact withthe first image displayed within the first frame; one or more graphicsprocessors to generate a second image for display within a second frameand a third image for display within a third frame, the first framehaving a first requested presentation time and the third frame having asecond requested presentation time; and a display engine to present thesecond frame for display at the first requested presentation time,present the third frame for display at the second requested presentationtime, and cause the display device to dynamically refresh to display thesecond frame and the third frame, wherein a refresh rate of the displayvaries between display of the first frame, second frame, and thirdframe.
 19. The display system as in claim 18, wherein the one or moregraphics processors are to generate the third image for display duringthe third frame before display of the second frame.
 20. The displaysystem as in claim 18, additionally comprising display logic toconfigure display events associated with the second frame and the thirdframe, the display events including configuration data to configure thedisplay engine to display the second frame and the third frame.